In some virtual reality and mixed reality display systems, it may be desirable to simulate immersive, touch-like interactions in the virtual environment. For example, a user may desire to manipulate a virtual object with her hand's motion in a manner similar to how she would handle such an object in the physical world. However, enabling such an immersive and natural touch-like interaction has proven challenging.
Previous attempts to enable interactions with a virtual object include using a symbolic or rendered representation of the user's hand. Such approaches, however, typically utilize precise hand tracking which may add significant latency to the experience. Additionally, in such direct physical-virtual interactions the interacting hand is viewed “directly” by the user, such as via a transparent head-mounted display/virtual reality system. Such systems typically include significant virtual display latency, since the time of the user's direct viewing of the hand corresponds to the virtual system's processing just beginning. Accordingly, in these systems the display frame-rate (e.g., frames-per-second) is an unavoidable lower-bound for the experienced interaction latency. Viewing a symbolic or rendered representation of a user's hand also creates a less than intuitive and natural experience. Gesture recognition systems also typically utilize precise hand and finger tracking, and are restricted to a symbolic gesture “language” which the user is forced to learn.
Capturing and using a silhouette of the user's hand generally involves a user raising his hand toward a capture device and display screen, and performing movements in this raised position. In these systems the camera position (e.g. frontal sensor) can usually capture only the inner-palm when the hand is raised upwards (towards the sensor). Such a position may be tiresome and uncomfortable over longer periods. Further, the user's hand may obscure the user's view of the object being manipulated. Additionally, the captured image of the inner side of the user's palm is typically presented as a colored silhouette or shadow of the opposite, outer side of the user's hand. Such a manipulated representation of the hand also creates a perception of a less than intuitive and indirect interaction experience.
Another disadvantage of this kind of inner-palm silhouette presentation occurs when the hand changes distance from the sensor (e.g. the hand moves closer to virtual objects and farther from the user, when reaching towards the display/sensor). In this situation, the size of the silhouette changes accordingly (e.g. becomes larger due to moving closer to sensor). However, to the user such change is counterintuitive as to how the silhouette would naturally change in order to realistically emulate the silhouette of the outer-palm (e.g. it would become smaller as it moves further from the person). In this manner, such silhouette-based systems provide less than realistic hand-image presentations.